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ABSTRACT 

The Illinois system has been designed to continuously 
evaluate the total occupational programs of its LEAs, Consisting of 
three phases, the system involves local planning, state review and 
evaluation of local planning documents, and on-site visitation to the 
LEA conducted by a team of individuals from outside the agency* The 
structure of the system also allows the State Education Agency staff 
to monitor any changes in local programs. There are several aspects 
of the system which have greatly enhanced its success: (1) the total 
occupational program, (2) the composition of the visitation team, (3) 
the fact that all team members have input to aspects of the final 
report, (4) suggested solutions to accomplish recommendations made by 
the team, (5) the nature of the Summary Conference which is designed 
to eliminate errors in the report before printing, and (6) the 
built-in follow-up of the system being the One-and Five-Year Plan for 
Vocational and Technical Education. (Author/CK) 
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The urgency of collecting evaluative data to substantiate and to 
aid in decision making is surfacing within our educational structure* 
Many State Education Agency (SEA) and Local Education Agency (LEA) 
personnel have recognized the need to evaluate and have expressed the 
willingness to become involved in needed evaluative efforts- Yet 
leadership in evaluating local programs has in many instances been 
neglected by SEA officials and left almost entirely to LEA personnel. 
The Vocational Education Amendments of 1968 — Public Law 90-576 — include 
a specific requirement that states evaluate the vocational programs 
which they support. States throughout the nation have responded to this 
mandate with varying degrees of enthusiasm, ranging from modest to total 
commitment. 

In Illinois a specific administrative unit has been established 
within the Division of Vocational and Technical Education and given the 
responsibility for program approval and evaluation. Emphasis of this unit 
was placed on the development, refinement, and maintenance of complete and 
total occupational programs — greatly different from the traditional emphasis 
on supervision of subject matter areas. 

Illinois has over 600 LEA's which offer occupational programs and 
consume over 30 million dollars of state and federal funds annually. To 
encourage relevant total program planning and development, LEA's have 
been given the autonomy to individualize their programs to meet the needs 
of their students and community. A state-administered system was developed 
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to monitor and evaluate these divergent local programs of occupational 
education. The Illinois system, the Three Phase System for Statewide 
Evaluation of Occupational Education Programs, has the following goals: 

1. To promote and assist with the development of quality 
occupational education programs at the local level. 

2. To assure accountability of federal and state funds 
allocated to the local occupational education programs. 

3. To provide the Illinois Division of Vocational and Techni- 
cal Education and other state agencies (Illinois Junior 
College Board, Board of Higher Education, Office of the 
Superintendent of Public Instruction) with data upon which 
statewide planning of occupational education can be based. 

The design of procedures to meet these goals has been characterized 
by a series of developmental stages involving hundreds of individuals 
from diverse backgrounds. (We believe this has contributed immeasurably 
to the acceptance and success of the system. ) 

Development began with several SEA-conducted workshops which 
included representatives from: business and incastry, secondary schools, 
community colleges, universities , and the state advisory council. These 
workshops were held to gain input, from various groups regarding the needs 
as well as procedures for evaluation. Following the initial planning 
workshops, a mail survey method--the Delphi technique — was utilized to 
determine the objectives of occupational education and to define a total 
program in the state of Illinois. A total program, as conceived in 
Illinois, has eight components or Areas of Concern. These Eight Areas 
of Concern will be described in detail later in this paper. Briefly, 
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administrative organization, personnel, obiectives, evaluation, occUDa- 
tional programs, resources utilized, guidarice services, and students served 
were identified as being components of .-i total orogram. 

The actual evaluation system was then designed around the three pre- 
viously mentioned goals and the Eight Areas of Concern. A field test of 
the system was made in seven representative LEA's during the 1970-71 school 
year. The field test led to many revisions m procedures and instrumenta- 
tion and the revised system was implemented in 71 districts during the 
1971-72 school year. At the close of that school year an additional work- 
shop was held, which included the leaders of on-site visitation teams, 
for the purpose of further refining the system. For the current year the 
scope of the system has been expanded to include the evaluation of 115 LEAs 

The central theme of the evaluation system is the analysis of the 
lea's total occupational program. This is accomplished through three 
phases — a local planning phase, a phase in which local plans are evaluated 
by SEA staff, and an on-site evaluation phase. 

PHASE I 

To implement the Vocational Education Amendments of 1968, Illinois 
granted autonomy for the administration of occupational education to LEAs. 
The first phase of the Illinois evaluation system involves staff of the 
LEA; their task is the planning of an occupational program best suited to 
their students. The key to Phase I 1-^ the total program planning conducted 
by the LEA staff. To aid local agenrdes in the planning process and to 
provide a uniform format for reporting, a guide for preparing Local Agency 



One and Five Year Plans for Vocational and Technical Education (D^^E, 
1971a) has been developsd. The Local Agency Plan, completed and submitted 
c^nnually to the SEA, is essenrially a planning docuinent:, and also serves 
as a contractual agreement betv;een the LEA and the SEA. 

In addition to describing the projected individual occupational 
programs, the Local Agency Plan outlines the total occupational program. 
Identified by the plan are: the administrative organization of the LEA, 
a listing of personnel, the LEA's program objectives, a description of 
the lea's evaluation system, a description of program offerings, a descrip- 
tion of how the community and school resources are utilized, and a 
description of available ancillary services. 

PHASE II 

Following its preparation and subsequent review by the local goverziing 
board, the Local Plan is forwarded to the SEA. During Phase II, Regional 
Vocational Directors m the Division's Program Approval and Evaluation 
Unit review and evaluate each Local Plan and recommend an approval status 
to the State Director for the occupational programs within each LEA. 
Each LEA is considered separately as it strives to achieve its potential 
within Its own surroundings- Such things as past performance, population 
characteristics, and available training opportunities are taken into account 
during this phase.. Phase li is complete when the Local Agency Plan is 
given an overall approval status by the SEA* 

Phases . I and il of the Evaluation system are completed annually. Phase 
I involves the annual preparation of the Local Agency One and Five Year 



Plan for Vocational and Technical Education, and Phase II involves the 
annual review and approval of the cocuir.snt by SEA personnel. The content 
of the Local Agency Plan is vital no the third phase of the evaluation system. 

PHASE III 

Phase III consists of on-site evaluations in a selected number of 
LEAs each year. The Illinois system concerns itself with the evaluation 
of the total occupational program as it exists within the LEA's surroundings. 
Eight total program components have been identified as being important 
to the S'-^ccess of a program in tems of meeting the needs of students. 
Thesfe components are assessed in terms of their contribution to a successful 
total program. A description of the eight components is given below. 

Administrative organization is one component which contributes to 
the success of a total program. The term does not refer to the adminis- 
trative staff per se, but instead refers to the organizational structure 
within which the staff functions to conduct all occupational education 
within the LEA. 

A second component involves the personnel involved in the total occu- 
pational program of the LEA. Personnel includes administrators, instructors , 
counselors f and other individuals who provide services for occupational 
students. The staff's qualifications, v/orking relations with other profes- 
sional' staff and with individuals in the community, as well as their atti- 
tudes and capabilities with regard to training students for eT^ployment 
are analyzed in the assessment of this component. 

A third factor important to the success of the total occupational 
program is the extent to which comprehensive and measurable obj ectives 
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have been established for the total program The ob:]ectives component, 
involves objectives de\''eloped for the total program, for mdJ'^idual 
occupational areas, and for individual courses. A portion of the 
evaluation process consists of an analysis of the appropriateness of 
these mc-.isurable ob:3ectives. 

A fourth component, evaluation , is directly related to objectives. 
Assessment of the LEA*s system for evaluating the occupational program in 
light of its established objectives comprises analysis of this component. 

The fifth component, occupational programs , involves a cursory review 
of programs in relation to the needs of the student population. The 
occupational programs coirponent possesses an interrelationship with 
each of the other conponents. 

Resources utilized is the sixth factor to be considered in the 
operation of a total occupational program. This includes th^?. utilization 
of in-school resources, advisory committees/ and other community agencies 
and resources (facilities and personnel). 

The seventh and eighth factors are guidance services and students 
served. Like objectives and evaluation, these are closely related. 
Effective guidance services often directly determine the extent to 
which student needs and interests are identified and met. Emphasis is 
placed on determining the extent to which disadvantaged and handicapped 
students are being served by the LEA's programs, and on determining the 
role of student organizations in the program. 
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Staffing 

On-site evaluati-ons are conducred by a team of individuals, 
selected by the SEA, who are capable of assessing the eight component 
parts mentioned above. Each team 15 headed by an individual who 
possesses broad knowledge of occupational education and has participated 
in an intensive training session conducted by SEA staff. The 25 team 
leaders for the 1971-72 school year, and the 45 leaders during the 
1972-73 school year included representatives from business and industrial 
firms, lea's, and universities- Each teaia leader conducts three 
evaluation visits during the school year. Team members comprise repre- 
sentatives from three groups — educators, business and industrial personnel, 
cind students- To date, educator team members have included personnel 
from secondary and post-secondary institutions; administrators, counselors, 
and teachers- Business and industrial representatives have been 
selected from varying types of backgrounds (industrial, service, agri- 
culture, etc-), and were often members of advisory committees or 
school boards from other LEAs- Student team members have been selected 
from secondary schools, post-secondary schools and business or industrial 
firms which have employed recent graduates of an occupational program - 
The number of team members for each school is dependent upon the size 
of the educational agency being evaluated- During the past two years, 
teani size has ranged from four to twenty-four individuals. 



Pre-Visitation Activities 

In advance of a gxven visitation, the chief school administrator 
of cui LEA to be evaluated is notified of the visitation via letter* 
Along with outlining the evaluation activities, the letter requests the 
administrator to inail a packet of information regarding the LEA to 
each of the team members who have been chosen for the visit. The 
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packet includes the Local Agency One and Five Year Plan for Vocational 
and Technical Education, and demographic data on the school and commu- 
nity. This packet gives each team member a chance to acquaint 
himself with the activities of the LEA prior to the on-site visit. 

Approximately five weeks before each visitation, the Regional 
Vocational Director ^ the SEA representative for that particular 
region, holds an evaluation orientation session within the LEA for 
all personnel associated directly with the total occupational program 
(administrators, instructors, and guidance personnel). This orientation 
has been simplified with a standardized slide/tape presentation to 
be followed with a question and answer period. Also during the orienta- 
tion session, a questionnaire (DVTE, 1972b) is administered to each ind;L- 
vidual present. This instrument has been designed to gather inf otta^iation 
concerning the local program from those who conduct that program. 

The SEA staff member also administers a questionnaire to a s.ample 
of the occupational education student body. The results of the 
student and staff questionnaires are summarized by computer, duplicated, 
and mailed to each team member prior to the visitation. The questionnaire 



results, along with the material mailed to the tean: members by the LEA, 
provide the opportunity for each team member to establish a uniform 
baseline of information to build upon during the visit. 
Visitation Activities 

On-site visits, depending upon the number of attendance centers 
within the LEA, last either two or three days. The evening prior to 
the visitation, all team members meet for an orientation session 
conducted by the team leader. The philosophy and mechanics of the 
evaluation, and the available information are all discussed during this 
team menber orientation. 

The following morning, the first day of the visit, a meeting is 
held for team members and all staff associated with the total occupa- 
tional program. The purpose of this meeting is to introduce the 
evaluation team and to allow the local administrator or his designee 
to give a short presentation pertaining to the LEA's program offerings, 
resources, and plans for the future. Following this meeting, the 
evaluation team begins interviewing individuals within the school and 
community according to the schedule established by the team leader. 
Interviews are structured through the use of a Team Member Handbook 
(DVTE, 1972c) , divided in accordance with each of the eight components fo 
evaluation, which identifies definite objectives that should be achie' -^d 
by the team member as a result of the interviews. Since the emphasis of 
the evaluation is on the total program, each subject matter area is not 
evaluated by an individual from that s;pecialty area. Instead, each team 
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interviews a crossection of the occupational staff, academic 
otaff, students and related personnel. In addition, advisory committee 
members, board of education members, and other community representatives, 
are interviewed to gain as many different vantage points to the local 
program as possible. 

Preparation of the Evaluation Report . 

Following the first day's interviews within the community and the 
educational agency, the team leader moderates a discussion of the day's 
findings organized around the Eight Areas of Concern. The following 
day or days, depending on the length of the visit, are spent interviewing 
and filling information gaps or voids which are identified during the 
team discussion. 

The team leader has the responsibility of summarizing the discussions, 
and with the team's aid, placing the results in the format of the evaluation 
report which includes a section for each of the eight componen1^,.;_Xor_ 
each area a series of conclusions indicating strengths and deficiencies 
are listed. For each conclusion which identifies a deficiency, a 
recommendation and a number of suggested solutions for correcting the defi- 
ciency are listed. Prior to leaving the LEA, the tea.m agrees upon the 
content of the entire report. 

The LEA personnel receive the findings of the team in both verbal 
and printed form. The visitation culminates in the presentation of the 
team's findings and suggestions to- a selected group of individuals responsible 
for administering and guiding the LEA's occupational program. A Summary 
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Conference moderated by one of the Regional Vocational Directors is held, 
with the team leader presenting the evaluation report verbally to a 
group of individuals comprising the Chief School Administrator, school 
board members, advisory committee members, and representatives of the 
instructional and administrative staff- The Summary Conference provides 
an opportunity for the local representatives to di^scuss the findings 
with the team leader, and ensures that any unsubstantiated statement 
in the report may be r^^vised by the team leader prior to printing. ' 
Following the Summary Conference the team leader transmits the report 
to the SEA for typing, duplication and distribution. Within three weeks 
following the visitation, copies of the report are mailed to the LEA's 
chief school administrator in the quantity he has requested for distri- 
bution. 

All on-site evaluations during the school year are conducted 
before the LEA submits its next annual One and Five Y^ar Plan for 
Vocational and Technical Education (Phase One) to the SEA. This is 
critical to the Three Phase Evaluation System, since the next Plan 
provides a built-in follow-up of the LEA's activities following the on-site 
evaluation. Review of this Plan (Phase Two) in light of the recommenda- 
tions in the Evaluation Report enables SEA personnel to monitor progress 
in the LEA toward achieving any recommended improvements. If action is not 
apparent on the part of the LEA to correct identified deficiencies, action 
concerning funding of programs is taken by the SEA. LEA autonomy carries 
with.it the responsibility of accountability for program offerings. 
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STATE LEVEL EVALUATION 
In addition to general recommendations and suggested solutions 
for LEAs, local evaluation provides input for state level evaluation. 
The need for evaluation at the state level is evident. Discrepancies; 
or weaknesses which occur in most or all LEAs may indicate weakness 
in preservice or inservice training for educators, rather than local 
deficiency. The data collected from each of the schools visited is 
compiled and summarized to determine the status of occupational education 
within the state of Illinois. Data obtained from the One and Five Year 
Plans / the School and Community Data Form, the Pre-Evaluation Instrument 
results/ and certain aspects of the local Evaluation Reports are 
summarized by computer. Summaries of the year's data are provided each 
of the team leaders for their review. A three-day workshop is held in 
late spring, involving each of the team leaders. The purpose of this 
workshop is to prepare a Composite Evaluation Report (DVTE, 1972d) and to 
refine the evaluation system. The statewide evaluation report is completed 
with a format similar to the local agency report, with conclusions, 
recommendations, and suggested solutions. The report is written with 
three audiences in mind: the SEA, institutions of higher education, and 
LEA personnel* The report points up actions which should be taken by each of 
the groups in an attempt to remedy the statewide weaknesses. The state 
level evaluation report serves as a planning document in the same manner 
as the local report serves the local agency. 
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RESULTS OF THE EVALUATION SYSTEM 
Although the on-site phase of the Evaluation System wa*^ only fully 
implemented during the 1971-72 school year, review of subsequent One- 
and Five-Year Plans for Vocational and Technical Education has pointed up 
considerable impact on local programs in the evaluated districts. The 
One- and Five-Year Plan is a built-in follow-up to each evaluation, and 
is probaMy the single most important factor in ensuring the success . 
of the evaluation system. .rc^me brief examples will help to illustrate 
the types of changes the system has brought about in terms of the 
objectives that were established for it. 

Objective 1: To promote and assist with the development of quality 
occupational education programs at the local level. 

1- The involvement of studemts, community personnel, and LEA 
staff in planning. programs has markedly increased in the LEAs which were • 
evaluated during the past school year. This program planning has centered 
around the development of the One- and Five-Year Plc.a? and involvement in 
planning has greatly increased the awareness of the concept of a total 
occupational program within the LEA. 

2. Serious consideration has been given by many LEAs to the 
administrative organization for occupational programs. The result in many 
cases has been total reorganization, appointment of a local vocational 
director with sufficient time to coordinate the programs, or clarification 
of the job description of the person called vocational director. 
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3. It is becoming apparent that evaluations have acquainted many 
administrators for the first time with the concept of total occupational 
programs. Involvement of Board of Education meftibers in the evaluation 
has also increased their knowledge of the importance of the total 
program. 

4. Increased utilization of community resources has become 
apparent, as the advisory committee concept has expanded- In addition, 
the utilization of community people and facilities in instruction is 
expanding. School personnel are discovering that community people are 
willing to help in the training of students for employment. 

5. Locally-directed evaluation systems, heretofore usually 
non-existent, are being developed in many LEAs. In addition to 
comprehensive follow-up studies of students and their employers, the 
systems are incorporating evaluation by advisory committees, community 
representatives, students, faculty, and administration. Most systems are 
being developed around the concept of assessing measurable objectives. 

6. A noticeable change is occurring in occupational guidance 
services available to students. Through restructuring of guidance 
departments, more students of the LEA are being served. New awareness 
of occupational program offerings on the part of guidance staff 
members has helped in the placement of students in these programs. In 
addition / the implementation of formal job placement services has 
strengthened many guidance departments. 



15 



7. New awareness on the part of instructional staff members concerning 
the total program as opposed to their own specific program will have 
substantial impact on unified occupational offerings in the future. 

8. One indirect effect of the evaluation system has been the 
inservice education experience of those who have served on evaluation 
teams. The effect of this has been tremendous on the growth of involved 
staff members. In addition, the interviews with over 1,000 btisiness and 
industrial persons during evaluations have increased their awareness of 
the programs available to students. 

Objective 2: To assure accountability of federal and state funds 
allocated to occupational education programs. 

1. The broadening of occupational program offerings available to 
students has been apparent. In addition to program expansion within 
LEAS/ more programs offered by Area Vocational Centers have been made 
available) to students/ and joint agreements between two or more LEAs 
have further expanded program offerings. Joint agreements are ideal 
for LEAs which are not presently served by an area vocational center and 
are unable to offer a comprehensive program. 

2. The importance of the built-in follow-up to each evaluation 
cannot be ^stressed too much. LEAs are asked to respond in writing to 
the recommendations of the evaluation team in their next One- and 
Five-Year Plan for Vocational and Technical Education. This past year, 
13 of the 71 districts did not comply with this request. This resulted 
in the non-approval of their programs. With subsequent revisions / 
approval was gained by these 11 secondary and two post-secondary 
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agencies. Some non-occupational programs and co^^rses which were 
previously approved were not approved in the subsequent Phase II. Other 
programs which were not previous Iv Glai:-:ied by ^ '> ressived apprxjval 

and reimbursement as a result of the evaluation. 

Objective 3: To provide the Division of Vocational and Technical 

Education and other state agencies with necessary data 
upon which statewide planning of occupational education 
can be based. 

1. Consultant services available from the Division of Vocational and 
Technical Education have been reorganized to meet the needs identified by 
the evaluations. 

2. The Composite Evaluation Report (the compilation of data gathered 
from the evaluations) serves as a document for statewide planning of 
Division activities. Such activities include establishing priorities for 
research and development, professional and curriculum development, etc. 

3. Pre-service and inservice education provided by state universities 
are being altered to compensate for identified personnel deficiencies. 

The impact of the evaluation system has been tremendous in Illinois. 
The success of the system can be attributed to the fact that the develop- 
ment occurred through a process of mass involvement of persons throughout 
the state. 

META-EVALUATION 

Planning and evaluation activities are not only important to 
improving educational programs, but are also essential to the refinement 
of a statewide evaluation system. The Illinois evaluation syst^em was 
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evaluated (Wentling and Klit, 1973) thrc\^ ' 5 of 0VQr.:\l techniques; 

^ j^ndividuals in contact with the system, am analysis of " 

the system by team leaders; , and a long-range study of t±ie impact 'of the 
system on Local Plans. A follow-up to the evaluation -iJas been, conducted 
involving individuals who have either been a part of tirs: s /stem or have 
been affected by it, including ' teachers , administrators lasam members, 
team leaders and SEA Regional Vocational Directors. The follow-up was con- 
ducted through utilization of a series of questionnaires jiesigned for the 
specific group being sampled. The questionaaires help ss^aluate team leaders, 
ideEEtify existing weaknesses in the mechanics of the E^csiiLuation Systentr 
and help determine the reliability and face validity of rhe system. 

The results of the follow-up are analyzed by the gromp of team leaders 
and, in addition to their personal experiences with ths system, are used 
collectively to evaluate the systejn, .resolting-liii writsisi ccmclusions , 
/recommendations and suggested solutions. This evaluation report is directed 
to the SEA staff to be used in revising and making imprcsvements in the 
evaluation system for future implementation. 

The impact of the evaluation system is assessed by the SEA staff 
members as they observe changes in the LEA's annual pl^ and, of course, 
actual changes and improvements in local programs. 

SUMMARY 

The Illinois system has been designed to continuously evaluate the 
total occupational programs of its LEAs. Consisting o± -insure phases/ 
the system involves local planning, state review and evalu^ifcLon of local 
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planning dociaments, and on-site visitation to the LEA conducted by a 
team of individuals from outside the agency. The review of planning 
documents, as well as the on-site visit, provides vital feedback to the 
LEA regarding the improvement of its total occupational program. The 
structure of the system also allows the SEA staff to monitor any chanaes 
in local programs through review of the Local Plans. This interlinkage 
of phases is an important and outstanding feature of the Illinois 
Evaluation System. 

Essentially, there are several aspects of the system which have 
greatly enhanced its success- These include the concept of 1) the total 
occupational progrJain with its eight important component parts, 2) the 
composition of the visitation team which includes students and employers 
as well as educators, 3) the fact that all team members have input to all 
aspects of the final report, 4) suggested solutions to accomplish 
recommendations made by the team, 5) the nature of the Summary Conference 
which is designed to eliminate errors in the report before printing, and 
6) the built-in follow-up of the system being the One- and Five-Year 
Plan for Vocational and Technical Education, All these factors have 
combined to make a meaningful, understandable evaluation system which 
works in Illinois. 
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